There aren't any python-confluent-kafka releases in the 1.x series that have Python 3.12 packages on conda-forge (https://anaconda.org/conda-forge/python-confluent-kafka).

Resulting in build errors like this:

  LibMambaUnsatisfiableError: Encountered problems while solving:
    - package python-confluent-kafka-1.9.0-py310h5764c6d_0 requires python_abi 3.10.* *_cp310, but none of the providers can be installed
  
  Could not solve for environment specs
  The following packages are incompatible
  ├─ python-confluent-kafka >=1.9.0,<1.10.0a0  is installable with the potential options
  │  ├─ python-confluent-kafka [1.9.0|1.9.2] would require
  │  │  └─ python_abi 3.10.* *_cp310, which can be installed;
  │  ├─ python-confluent-kafka [1.9.0|1.9.2] would require
  │  │  └─ python_abi 3.7.* *_cp37m, which can be installed;
  │  ├─ python-confluent-kafka [1.9.0|1.9.2] would require
  │  │  └─ python_abi 3.8.* *_cp38, which can be installed;
  │  ├─ python-confluent-kafka [1.9.0|1.9.2] would require
  │  │  └─ python_abi 3.9.* *_cp39, which can be installed;
  │  └─ python-confluent-kafka 1.9.2 would require
  │     └─ python_abi 3.11.* *_cp311, which can be installed;
  └─ python 3.12**  is not installable because there are no viable options
     ├─ python [3.12.0|3.12.1|...|3.12.5] would require
     │  └─ python_abi 3.12.* *_cp312, which conflicts with any installable versions previously reported;
     └─ python 3.12.0rc3 would require
        └─ _python_rc, which does not exist (perhaps a missing channel).

(build link)

Pushed a commit (8c206d7) which updates python-confluent-kafka and librdkafka to the latest minor version (the 2.5.x series) and continues the practice to pinning build and runtime dependencies within a single minor version of Kafka libraries.

Updating the Kafka dependencies is fine with me.

cc: @jdye64 for awareness.

The use of ast.Num here resulted in a deprecation warning that was treated as an error in tests (thanks @mroeschke !).

DeprecationWarning: ast.Num is deprecated and will be removed in Python 3.14; use ast.Constant instead

(build link)

I looked into this... seems ast.Num and other specialized constants have been deprecated since Python 3.8 and are set to be removed in Python 3.14.

This proposes checking the value of the ast.Constant being passed in here against a hard-coded set of builtins.

• more details on that deprecation: https://github.com/python/cpython/blob/aa1339aaaa6363c38186defaa079d069b4cb08b2/Doc/whatsnew/3.8.rst?plain=1#L1668-L1671
• evidence that this list of builtins is identical to what ast.Num and ast.Str were doing: https://github.com/python/cpython/blob/a50a86db0fd66c7ea53e56e346057be1153d6d47/Lib/ast.py#L528-L534

This looks correct to me. Thanks.

Starting with Python 3.12, the repr() representation for an OrderedDict has changed:

• OrderedDict dict/list representation python/cpython#101446
• https://docs.python.org/3/whatsnew/changelog.html

(thanks @bdice for pointing me to that changelog item!)

That's causing doctest failures like this:

E       AssertionError: 1 of 13 doctests failed for DataFrame.to_dict:
E         **********************************************************************
E         File "/opt/conda/envs/test/lib/python3.12/site-packages/cudf/core/dataframe.py", line 2362, in DataFrame.to_dict
E         Failed example:
E             df.to_dict(into=OrderedDict)
E         Expected:
E             OrderedDict([('col1', OrderedDict([('row1', 1), ('row2', 2)])),
E                          ('col2', OrderedDict([('row1', 0.5), ('row2', 0.75)]))])
E         Got:
E             OrderedDict({'col1': OrderedDict({'row1': 1, 'row2': 2}), 'col2': OrderedDict({'row1': 0.5, 'row2': 0.75})})
E         **********************************************************************
E         1 items had failures:
E            1 of  13 in DataFrame.to_dict
E         ***Test Failed*** 1 failures.
E         
E       assert not 1
E        +  where 1 = TestResults(failed=1, attempted=13).failed

/__w/cudf/cudf/python/cudf/cudf/tests/test_doctests.py:125: AssertionError

(build link)

This proposes just skipping those tests, here and in the docs for Series.to_dict().

In the cudf and dask_cudf tests on Python 3.12, there are a few dozen cases where DeprecationWarning like the following are treated as errors

DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).

(build link)

Looking through the logs, those all appear to be coming from code in boto3 / botocore that we don't control. There are no direct uses of datetime.datetime.utcnow() in this repo.

git grep utcnow

So this is proposing not failing tests when those warnings are raised.

Context on the deprecation of datetime.datetime.utcnow():

• https://blog.miguelgrinberg.com/post/it-s-time-for-a-change-datetime-utcnow-is-now-deprecated
• https://docs.python.org/3/library/datetime.html#datetime.datetime.utcnow

Is it possible to scope this more narrowly? I haven't looked at the error logs to see how widespread this problem is, but I wonder if we can use something like a local context manager to wrap the places where this is occurring.

Let's not treat this as a blocker for this PR -- but we might want to make it a follow-up task.

I'm not sure. It's coming from this codepath in botocore

self = <botocore.auth.S3SigV4Auth object at 0x7f6feb62f380>
request = <botocore.awsrequest.AWSRequest object at 0x7f700a3b3800>

    def add_auth(self, request):
        if self.credentials is None:
            raise NoCredentialsError()
>       datetime_now = datetime.datetime.utcnow()
E       DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).

https://github.com/rapidsai/cudf/actions/runs/10724599183/job/29742782194#step:7:4279

If it's possible to ignore warnings from specific dependencies, that'd be useful. I couldn't figure out how to do that in a few minutes of googling and docs reading.

If it's possible to ignore warnings from specific dependencies, that'd be useful. I couldn't figure out how to do that in a few minutes of googling and docs reading.

In a warning filter, after specifying the warning category you can also specify :module:line https://docs.python.org/3/library/warnings.html#describing-warning-filters

So in a follow up PR, you can add :botocore at the end to just ignore this warning for that module

Thanks @mroeschke , I finally went back and did this. You suggestion worked perfectly 😁

#16896

Updating the Kafka dependencies is fine with me.

cc: @jdye64 for awareness.

This looks correct to me. Thanks.

Is it possible to scope this more narrowly? I haven't looked at the error logs to see how widespread this problem is, but I wonder if we can use something like a local context manager to wrap the places where this is occurring.

Let's not treat this as a blocker for this PR -- but we might want to make it a follow-up task.